Traffic mode recognition is an important branch of user behavior recognition, the purpose of which is to identify the user's current traffic mode. Aiming at the demand of the modern intelligent urban transportation system to accurately perceive the user's traffic mode in the mobile device environment, a traffic mode recognition algorithm based on the residual temporal attention neural network was proposed. Firstly, the local features in the sensor time sequence were extracted through the residual network with strong local feature extraction ability. Then, the channel-based attention mechanism was used to recalibrate the different sensor features, and the attention recalibration was performed by focusing on the data heterogeneity of different sensors. Finally, the Temporal Convolutional Network (TCN) with a wider receptive field was used to extract the global features in the sensor time sequence. The data-rich High Technology Computer (HTC) traffic mode recognition dataset was used to evaluate the existing traffic mode recognition algorithms and the residual temporal attention model. Experimental results show that the proposed residual temporal attention model has the accuracy as high as 96.07% with friendly computational overhead for mobile devices, and has the precision and recall for any single class reached or exceeded 90%, which verify the accuracy and robustness of the proposed model. The proposed model can be applied to intelligent transportation, smart city and other domains as a kind of traffic mode detection for supporting mobile intelligent terminal operation.